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[57] ABSTRACT 
The disclosure is directed to an improved method and 
apparatus for generating a sequence of video frames 
representative of three-dimensional animation. A plu- 
rality of key frames are stored, each key frame including 
a common figure having one or more joints, and each 
joint having associated therewith a set of vectors defin- 
ing a umb. Each joint is defined in each frame by open- 
tor-controllable parameters which determine the three- 
dimensional position, rotational orientation, and scale 
factors of a local coordinate system in which the limb 
vecjorsjire placed. A plurality of in-between frames are 
generated, the m-between frames including the com- 
mon figure having one or more joints and limbs corre- 
sponding to the joints and limbs of the common figure 
in the key frames. The parameters of the joints of the 
m-between frames are obtained by interpolating in three 
dimensions, the position, rotational orientation, and 
scale factors of the corresponding joints of the key 
frames. In the preferred embodiment, the joints of each 
figure are arranged in hierarchical order, and the posi- 
tional coordinates and rotational orientations of the 
local coordinate system for a particular joint are deter- 
mined with respect to the local coordinate system of the 
next higher joint in the hierarchy. Also, the operator 
can control the interpolation during display of the in-be- 
tween frames, so as to change the motion of a figure 
limb. 

22 Claims, 11 Drawing Figures 
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THREE DIMENSIONAL ANIMATION ^ ^ ^ " 10 generate « series of 

at»uwaiiun animated frames. Particularly when dealing with three- 

BACKGROUND OF THE INVENTION dimensional animation, however, the automatic genera. 

_. . . non of in-between frames can tend to result in unnatural 

This invention relates to the field of machine genera- 5 motion of the three dimensional figures. 

B °°? m ° re particuiarly ' 'L" mcthod » »aoii« the objects of the present invention to 

ably high quality haveSonalry be«^y 52? appearance of motion of figures with mini, 
to make. The initial creative work includes generation 

of a basic story, a sound track, and basic character de- SUMMARY OF THE INVENTION 

signs. Typically, a schedule is generated which de- 15 tk- ^ J .» lt , • • . J . 

scribes the movements and timings of each scene to be SL^f^ invention is directed to an improved 

produced, and layout sketches and "key" drawings are m ™T md *PP* mua for generating a sequence of 

made which define the overall design of each it *° 681063 representational of three-dimensional ani- 

has been generally observed that at this point most of matioiL 111 accordancc wth the method of the inven- 

the creative design work has already been done. The 20 &QP * a plurali ty of key frames are stored, each key frame 

so-called "key- drawings or frames are still frames ^eluding a common figure having one or more joints, 

which show extremes of action. In the traditional ani- and 6800 J omt having associated therewith a set of vec- 

mation procedure, people known as "m-betweeners" ton defining a limb. Each joint is defined in each frame 

are employed to sketch frames which fill in the action by 0 Perator-controllable parameters which determine 

between key frames. After in-betweens are sketched by & me three-dmiensiona] position, rotational orientation, 

hand, the hand-drawn pictures are typically transferred ***** scale factors of a local coordinate system in which 

onto celluloid-acetate sheets and are then painted m by ue lunb vectors are placed A plurality of in-between 

hand. Much labor is involved in obtaining the in-be- frames are generated, the in-between frames including 

tweens, which typically comprise the majority of the common figure having one or more joints and limbs 

frames of the final work product These tasks render the 30 corresponding to the joints and limbs of the common 

production of animated films an extremely expensive figure in the key frames. The parameters of the joints of 

proposition. An advantage of the traditional animation the in-between frames are obtained by mterpolatinc, in 

process, however^ that the animator has virtually three dimensions, the position, rotational orientation, 

complete amsuc freedom and control over the resulting and scale factors of the corresponding joints of the key 

film; i*. anything that u drawn can be made to move in 35 frames. 

^tltv^ 80 natural - k,okin 8 ma6oa <*» In the preferred embodiment of the invention, the 

in r~*Z v«L tK* fi.M ~r w • * i ojnXS of each figure are arranged in hierarchical order, 

graphics tec^SK 

ity of computer-generated ammation. ^iWcan ^^^^/^ * ^ local , coordinatt 
aid artistic arumators in many ways, SuiTau^ ° fthe nc « higher jomt in the hierarchy. Also, in 

matic generation of in-between frames by mtenilation f ccordancc w * mc P refcrred embodiment, the opera- 
of lines appearing in successive key frames. A technique ^If" COnt ?° 1 the "^rpolation ***** display of the 
of this type is described, for example, in an article end- 45 ? -bctw . ec ? rramcs - so as to change the motion of a 
tied 'Towards s Computer Animating Production 

Tool- by N. Burtnyk and M Wein which appeared in w™"^" features 411(1 advantages of the invention will 
the proceedings of Eurocomp Conference, Brunei - become more apparent from the following detailed 
United Kingdom, May 1974*. description when taken in conjunction with the accom- 

It is known in the computer graphics art that three-di- 50 Panymg drawings, 
mensional representation of animated figures can be BRIEF DESCRIPTION OF thp nn awtmpc 
stored as a set of three dimensional points, and appropri- DESCRIPTION OF THE DRAWINGS 

ate transformations can be used to automatically com- FIG. 1 is a block diagram of an apparatus in accor- 
pute the projection of the three-dimensional figures dance with an embodiment of the invention, and which 
onto a two dimensional surface, so that two dimensional 55 0211 be used to practice the method of the invention, 
computer generated animation images can be presented FIG. 2 is a flow diagram which describes the general- 
on a conventional twoKlmiensional screen. Three di- procedure for producing three-dimensional anima- 

mensional figures have also been represented in joint uon on a two-dimensional screen, including the tech- 
and limb configuration. nique of the invention. 

Representative prior art patents relating to tech- 60 FIG. 3 is a flow diagram of the routine for forming 
niques such as computer-generation of in-between skeleton files for a sequence of animation to be gener- 
frames and three-dimensional animation are as follows: ated. 

U.S. Pat. Nos. 3,364482; 3,523,389; 3,385,623; FIG. 4, which includes FIGS. 4A and 4B, illustrates 
3,603,964; 3,700,792; 3,723,803; 3,747,087; 3,792,243; a joint and Hmb in a local coordinate system, and shows 
3,883,861; 3.917,955; 3,885,096; 4,017,680; 4,127,849; 65 the manner in which a joint and limb are represented 
4,189,743; 4,189,744; 4,200,867; 4,213,189. and can be nianipulated. presentea 

As above-stated, the automatic machine generation of FIG. 5 is a flow diagram which illustrates the routine 
in-between frames can greatly reduce -the amount of for displaying frames of figures represented by joints 



09/26/2002, EAST Version: 1.03.0002 



4,600,919 

3 * 

and bmbs in three dimensions on a two-dimensionaJ line drawing system 70, and includes processor 71 and 
d *Pky. line drawing display 72. A suitable vector processing 

FIG. 6 is a flow diagram of the routine for forming a and display system is the Evans and Sutherland Com 
frame, such as the key frame pany "Multi Picture System". A raster-based system 

FIG. 7 is a flow diagram of the routine for generating 5 which performs similar functions is described in an 
in-between frames. articJe entitled 'X^rproccssing to Ease The Graphics 

FIG. 8 is a flow diagram of a routine for modifying Borden" which appeared in the July, 1982 issue of 
the motion of a limb during display. "Computer Design**. The system 70 is coupled via the 

FIG. 9 shows an interpolation curve and a modified bus 30A to the other equipment A video tape recorder 
interpolation curve. 10 55 and a television camera 65, may also be provided in 

FIG. 10 is a flow diagram of the routine for modify* the system, 
ing the interpolation curves. Referring to FIG. 2, there is shown a generalized 

DESCRIPTION OF THE PREFERRED flow ^gram of a typical operational procedure for 

EMBODIMENT generating a sequence of frames of three dimensional 

IS a n imat ed scenes for display on a two-dimensional 
Referring to FIG. 1, there is shown a block diagram screen. It will be understood; however, that variations 
of an apparatus 10 in accordance with an embodiment in this typical procedure can be employed, consistent 
of the invention and which can be used to practice the with the principles of the invention, depending, inter 
method of the invention. An operator 15, who is typi- alia, upon the type of animation to be produced, the 
cally, but not necessarily, an animator of some artistic 20 source of component figures, and operator preference 
ability, works at a console which includes the devices The block 211 represents the inputting and storage of 
illustrated within the dashed enclosure 20. la the pres- limb files, which, if already existing, can be keyed in via 
ent embodiment, the console includes three types of the keyboard 22, or can be drawn in via the data tablet 
input devices whereby the animator can input inform*- 21, or by using the joysticks and cursor control. In the 
tion to the apparatus 10, viz. a data tablet 21, a keyboard 23 present embodiment, each limb is represented, in known 
22, joysticks 23 and monitors on which information is fashion, as a set of vectors in three dimensions which 
displayed, viz. monochrome display 23 and optional define the limb shapes to be used on the figures in the 
color display 25. sequence of animation. The form of representing the 

The devices in the console 20 are coupled via a bus vectors and limbs is treated hereinbelow. It should be 
30A to control circuitry 30. In the present embodiment 30 noted that this part of the procedure is not strictly nec- 
the novel functions of control circuitry 30 are imple- essary to perform at this time (since limbs can be pro- 
mented by an appropriately programmed general pur- duced after the skeleton of joints is formulated, as de- 
pose digital computer, for example the model VAX scribed bereinbdow), but in many practical situations 
1 1/780 or the model PDP-1 1/45 manufactured by Digi- one can utilize previously stored three dimensional 
tal Equipment Corp. of Maynard, Mass. However, it 35 vector representations of limbs available from a M limb 
win be understood that alternate means, such as a spe- library". Also, it is usually convenient to generate limbs 
cial purpose computer or other suitable circuitry having beforehand for use during the sequence of animation to 
logic and storage capabilities could be utilized to be generated, it being understood that certain limbs can 
achieve the desired functions. In conjunction with the be utilized a number of times in the various figures of 
general purpose digital computer 30 there is provided 40 the animation- The representative limb to the left of 
memory which includes fast access bulk storage such as block 211 is a simple robot foot 
disk memory 52, and random access storage labelled The block 212 represents the formation of a skeleton 
RAM S3. Typically, at least a portion of the random file for each figure in the sequence of animation to be 
access memory will be included within the general generated. As previously indicated, each figure includes 
purpose computer 30, and it will be understood that the 45 one or more joints, and each joint has associated there- 
amount and type of each storage medium or alternative with a limb. Each joint is defined in each frame by 
storage can be selected by one skilled in the art. operator-controllable parameters which determine the 

An optional frame storage means 60 is provided and three-dimensional position, rotational orientation, and 
is coupled, inter alia, to the control circuitry 30 and to scale factors of a local coordinate system in which the 
the color monitor 25. The frame storage means is of the 50 limb vectors are placed. The joints of each figure are 
type known as a "frame buffer" manufactured and sold arranged in a hierarchical order, and the positional 
by Genisco Computer Company of California. Essen- coordinates and rotational orientations of the local co- 
tially, the frame buffer 60 is an addressable digital mem- ordinate system for a particular joint are determined 
ory which stores a frame of video information. The with respect to the local coordinate system of the next 
frame buffer 60 is coupled to the control circuitry 30 via 55 higher joint in the hierarchy. The information defining 
the bus 30A so that the control circuitry 30A can typi- these parameters for each joint will be described in 
cally be utilized to address any desired pixel in the further detail hereinbelow. A simplified skeleton figure 
frame buffer, for example to interrogate or read the is illustrated to the left of block 212 of FIG. 2. In this 
pixel value contained therein or to write in a new pixel sketch, the joints are represented by small circles and, 
value at any point 60 for ease of illustration, straight lines are shown as con- 

In the present embodiment of the invention, it is con- necting each joint to its "parent" joint; Le, the joint of 
venient although not required, to employ a commer- next highest priority in the joint hierarchy. In the 
cially available processor for performing certain matrix sketch, the joint Jl is the highest priority joint for the 
transformations and display tasks which could al terns- figure, so the motion, rotation, or scaling of the joint 
tiyely be performed by general purpose processing and 65 parameters for Jl will serve to move, rotate, and scale 
display circuitry. In particular, a three-dimensional line the entire figure in the world coordinate system. The 
drawing display system with coordinate transformation joints J2, connected to joint Jl as their parent, are the 
and clipping hardware is collectively referred to as the • next lower order joints in the joint hierarchy. There are 
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three such child joints J2, designated as J2A, J2B, and 
J2C Also, the joint J2A is a parent of three more child 

joints designated as J3A, J3B, and J3C m, «,"| 0) 

The routine for forming and storing the figure skeie- r * r ' r ' 

tons is shown in FIG. 3. The block 311 represents the 5 L* * * J 
identification of the skeleton file for the particular fig* 

ure whose skeleton b to be generated. A joint is idenri- The vaiues m /' °V Md m * respectiveJy represent the x, 

fied to be input (Mock 312) and its connection m the y / a ? d 1 1***** 0 * or "nx>ve" coordinates of the origin of 

hierarchy is indicated, e.g^ by identifying its parent , n ^ coordinate *e Point at which the 

joint If desired, the skeleton can be displayed during Particular joint is defined as being located) with respect 

this operation (see display routine of FIG. 71 Initial x. y t0 . cl ^ cr .™ coordinate system (for the highest 

and z coordinates of the joint position are stored, as * mt \ or t0 0,6 !ocaJ system of the 

represented by the block 313. Initial rotational orient*- 15 « r '""J* m te hicrarch y of Jomts. The 

tion values and scaling factor values can then also be 13 l^'nr^^t * "S""" "ft* 0 *™ rot * tionai «- 

block314. TOemeanmga^pur^ofthesevahiesare joint coordinate system). The values s x , 
described further herembelow. IT* next joint of the „ scaling factors forAexfy and z direct of the?oc2 
figure can then be added (re^try to block 312). Addi- 20 coordinate system. A value ofwtCKteSS 
tional figure skeletons can be formed and stored by the normal scale factor in each dimension, and higher or 
repeating the procedure. lower numbers can be used to stretch or compress the 

Referring again to FIG. 2, the block 213 represents limb associated with the local coordinate system in any 
the portion of the generalized procedure wherein indi- 25 direction. 

vidua] figures of a frame are formed, so that the desired FIGS. 4A and 4B are useful in understanding the 
poses are obtained for the key frames of the animation manner in which the joints and limbs are represented 
sequence to be generated. This routine is described in manipulated. In FIG. 4A, there is shown the coor- 

conjunction with FIG. 6. The figure sketched to the left dinalc **« for &° exemplary joint, J* and a limb, U 
of block 213 is useful for initial illustration of how the 30 ( s ^ c > wn m dashed line) of vectors associated with the 
limbs, dr?Bgnatrd with notation similar to the joints with J oinL Tne matrix values for the joint J,* is as follows: 
which they are respectively assoriarrri, except that an L 
is used instead of a J, are stored in conjunction with the 
skeleton file. Thus, each joint of the skeleton file for a 
figure has stored in conjunction therewith a limb file (or 
identification of the limb file) for the particular limb that 

t^*r b Z % C ° PCnUOr f ° r W M * joint Assceni * TO- *A, the origin of the local coordinate 

position of the figure. system for j . ^ at ^ poiflt (10 w 1Q) m ^ Qr 

The key frames are stored (block 214) and, under 40 "world" coordinate system (which, in this example, is 

operator control in-between frames can be automata- the coordinate system of next higher priority in the 

caily generated (block 215) in accordance with the hierarchy). Also, the x, y, and z axes are aligned with 

principles of the invention, and as described in conjunc- the axes of the world coordinate system, so the relative 

tion with the routine of FIG. 7. During or after such rotation angles in the second row of the matrix are all 

generation of in-betweens, the sequence of «nim«t~i 45 0*. Finally, it is assumed that, for this example, the scale 

frames can be displayed, and modifications to the se- factors for the frame are aD unity. The limb I^associ- 

quence can be implemented, as represented by block atcc * the exemplary joint is represented by the 

216 and as described in conjunction with the routine of following vector list: 

FIG. 8. The final sequence of frames can then be stored M 

(block 218) and rSeral (bk>ck 219) by p^^S KU*<^C-L-Ufta-.l* 
propria te surface characteristics to the wire frame type 

structures that result from the described procedure. The list sets forth the vertices of the limb as defined in 

Appropriate color rendering can also be implemented, the local coordinate system (FIG. 4). A comma sepa- 

as is known in the art. One or more frame buffers (eg. 55 rates the points which are joined by lines, so some indi- 

60) and color monitor 25 can be used toward this end. In vidua! points may be listed more than once, 

this regard, reference can again be made to the book Reference can now be made to FIG. 4B which repre- 

44 Principles of Interactive Computer Graphics", as well tnc same J° mt -I/and limb L* but with the position, 

as to U.S. Pat No*. 4,189,743 and 4,189,744, assigned to orientation, and size and shape of the limb being differ, 

the same assignee as the present application, and to the 60 OTt by virtuc of modifications in the matrix values of the 

patents listed in the background portion hereof. examplary joint In particular, the matrix for the joint of 

The position, orientation, and scaling factors of the FIG * 48 " M foUows: 
set of local coordinates for a particular joint is defined 

with respect to the coordinate axes of the next higher « r 10 l5 l0 -| 

priority joint in the hierarchy of joints. In particular, a 45' o* o* 

parametric matrix defining the local coordinate space is L 1 1 2 J 
given by: 



35 
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In this case, the origin of the local coordinate axes (Le. t 
the joint position) is seen to be at the coordinates (10, 15, 
10), which correspond to the values in the first row of 

the matrix. Also, it a seen thai the y, z axes are rotated [x'y / i) = [x/r i] 

by 45* around the x axis as compared to the orientations 3 
of the y, z axes in the parent w world** coordinate system. 
Accordingly, the second row of values has a 45* rota- It can be noted that this transformation matrix affects 
tional angle indicated in the x column, and 0° rotational only the values of the x and y coordinates. Similarly 
angles indicated for rotation around the y and z axes, jq derived transformation matrices can be set forth for 
Regarding scaling, the limb of FIO. 4A is seen to be rotation about the x and y coordinate axes, 
doubled in length or ''stretched" along the z axis (only). A scaling transformation, for use to scale dimensions 
Thus a scale factor of 2 is indicated in the z column of m 6400 coordinate direction, is as follows: 
the third row of the matrix. It will be understood that in 
the present embodiment the representations of the Hmb 1* 
vectors can remain fixed during a sequence of anima- 
tion, with the changes in position, orientation, and size [** / * tJ - fry* i] 
and shape of the limb being achieved by modification of 
the parametric matrix of the joint with which the limb is 

associated. Also, when a modification in a parameter of where s* Sj, and s, are, respectively, the scaling factors 
a joint is implemented, it is seen that the limbs of all in the x. y and z directions. 

lower priority joints are effectively modified along with & translation, composite rotation, and scaling 
their parent joints. matrices are represented by [M], [RJ, and [S], respec 

It is known in the computer graphics art that geomet- 23 ^ y 'J^ mposite transformation matrix [Tl can be set 
ric transformations can be used in generating an image " ~* 
on a rwo-dimensional scr e e n of three-dimensional 
scenes. Reference can be made, for example, to the 
bool^Principles of Interactive Computer Graphics,*' This composite transformation matrix corresponds to 
referred to hereinabove. In the present invention, ma* 30 ^ e composite parameter matrix for the joint local coor* 
trix transformations are employed to transform the limb dinate system first set forth above at (1). 
vectors in a given local coordinate system into eqmva- To transform the vectors of a limb associated with a 
lent vectors in the joint coordinate system of next high- joint J a which has respective parent, grandparent, etc. 
est priority. The procedure is repeated for successively joints J«_ \, J tt -2* etc., into a main or world coordinates 
higher priority joint coordinate systems until the limb system (represe nt ed as Jo), the overall transformation 
vectors are expressed in terms of the world coordinate matrix, [T ft+ o] can be represented as the concatenation 
system. Finally, a perspective transformation is used to of tne various iocal matrix transformations, as follows: 
project the three-dimensional scene onto a two-dimen- 
sional surface (e.g. corresponding to the display screen 40 
surface). This perspective transformation depends upon wherc ^ ^^prs of the transforms and the joints 
the location, rotational angle and viewing angle of an correspond. 

imaginary camera whose position and other parameters Once expressed- in terms of the world coordinate 
are variable under operator control, such as by manipu- system, a known viewing transformation [V] can be 
latmg one of the joy sticks. 45 used to convert points in camera two-dimensional per- 

It is known that if a geometric transformation does spective coordinate system (x^ y c ) in accordance with: 
not deform an object it transforms, then it must be de- 
composable into primitive translation and rotation Uc*M*d«t*wru»lHV) 
transformations. As described, the inclusion of a scaling r.« • , 

t«i«rf rt «« a f«%« ««««« et»-t^Ki«* o-a J> 50 where [V] is known to be a composite viewing transfer- 

tranrformauon aUows stretching and contraction of ^ buUt up from t^lanons and 

limbs during motion. The matrix transformation which ™r T K - 7* V . " u 

uiuws uiuuig uwuwn. nc ™™ uwuiunnauuD w ™ cn rotations that are determined from the viewing parame- 

translates a point fry*) to a new point (x',y',z) is ^ ^ Wf U the resulting homogeneous cooVdinate. 
known to be: Reference can again be made, for example, to the previ- 

55 ously cited book and patents. The transformation to the 
appropriate two-dimensional coordinates (x, y) is 
thereby seen to be achieved by multiplying the points in 
the world coordinate system by the viewing transfor- 
mation and subsequent division by the homogeneous 
60 coordinate. 

. . . r.i_ . , Referring to FIO. 5, there is shown a flow diagram of 

where m x , m,and m, are the ^components of the trans a- a routilie £ cluing th e display of ^ threeSimen- 
oon or move m the x, y and z directions ^respectively, $ionaI ^ stored £ mcmo * m joint ^ Umb for . 
and the 1 in each fourth column is the homogeneous ^ gaicral & c mvo lves the obtainment of 

coordinate. 65 the two-dimensionaJ (x,y) coordinates of the figures to 

Rotation about the z axis (for example), through an be displayed (for each frame) by concatenation of ma- 
angle 9 can be achieved with the following transforma- trix transforms, as previously described, to obtain the 
tion: two-dimensional projections of the limb vectors in the 
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main world coordinate system as viewed with an oper»- the cursor. The wi*neier to he mrvwiu* • 

tor-selected imaginary amen. The twoKdimenS rou^or soSinTT^tifieJ aTSSL^T*!! 

projections are displayed oa the line drawing display 72 Wock614 22*5* represeated °>* the 

of system 70. The set-up and multiply c^thTJj? E^cttTS^?^ 

ces can be unpfcmentai m the general purpose proces- 5 parameter, and a joy^TSeX S2m , 

^ 30 b"t. mo« preferably. » thieved by providing control the selected^r^^Hc^ S ^ 

Se^tnfp™ * *~ ^ 70 WWCh U ^ -^thau ifd^^rSScaL^J: 

In FIG. 3, theXTsn repress the reading of the ^^r^e^r^lSf ° f 
stored parametric mat™ ofthe next highest pnon* ,o ntu^seS^w o^^eSc^t 

joint (beginning wrth the highest priority joint). In the metric matrix (whSi^««M^^^^; 

matrix ttnfcMn is stored, as represented by the (block 618). In this hZHc sSJKTto f£»« 

block 313. It wul be understood that when the next can be formulated ands^d Y 

lower priority joint is processed, the stored coacatena- FIG. 7 illustrates the routine of the D r~nt ™^ 

bodinent the operator can elect to display either limbs produce natural I «otte Vl£Z£5ttF 

which comprise the actual figures, or skeletons which ^^m^T^^tlU^^L^^^ 
are useful m visualizing the position, of the joints and 30 m^TfSL ISsSSS t£S^ 

the statuses of the coordinate axes Hpfity^ t>v the joints. trmit^H ~ ^^JT/vi^L^f^^T' ^ 

If the figure limb, themselve. «rb7d£^cK STin^oS U^S ^rete^T^ 

2^- 'h "f^ thi !; b,OCk block vT^of the^^ ro^SoTS £ll 

w r^.««>^«ed ^ thecur. panuneter, ofthe tna^Sm^awSSS 
rent joint, and the multiplication of each point by the 35 joint are interpolated in the mTm^LTZ 

Z^^TS* ^nnanon. If th, figure sxele- > m . y'an? ,252.512 

tons are to be displayed, a base orthogonal coordinate x, y or x component is selected (Nock72» l3£. el 

3^„^ 1* C °_ n " tenWed ""I"* f« Ae selected component. In an operational 
SSSSJS to properly onented *> embodiment hereof, a cubic curve was fit through^ 

scaled local coordinate axes representation at each joint points in a standard in-betweenine nlM 0 ffL™„™£' 

potion, as shown in the small sketch near block 516. versus the JSSSSJSSffi £ gT£ 

The block 521 is then entered (from block 515 or 51«), 10). Block 723 is then ente^edTto *tock w^endna 

S^d ra ^^l!v^r ge ,^^ ° fraUla * e «* value, from ^t^la^^veln^e 

Zl^^A ^ y !?«°£ yStem ? A <»«ennina- 43 transformation matrices of the frames being in- 

non s then made (diamond 522) as to whether or not the betweened. In particular, the selectedone ofthe nine 

tat jaw : ■ the frame ha. been processed. If not. the values in the parametric mtrSleacn^ aboveT 

rW 'rftS^ * eframecaobe r T ! *i ed ' °. r t«e ■« based on the value of such parameter component taken 

!eX^M ^ ependm8 Up0n * e ftmctk>n ^8 30 from the cubic mterpoUuoT curve. Determination is 

R^riT^^Pir n „ _,. , then made as to whether or not all three components 

Referring to FIG. 6. there is shown a How diagram of have been considered (diamond 7241 If noTthe loon 

routine for forming and modifying the key frames, as 731 continues. When complete, a d«ermmatio« £ mao> 

represented generally by the block 213 of FIG. 2. The (diamond 725), as to whettoor notaTp^eten^e 

n%^^ f T^ 8 ^ rfW ^ <U)lMdtte 55 ^ Processed- "not, the loop 732 con^S 

k^ rW^/^ m -^^ r ? reSe ir lbythe P^etcrs hav e been processed, determination « made 

«w«Tn^W u S***? 1 " V C ^ hng up ,ts ( dian,ond '26). « to whether or not all joints for all 

skeleton file (and the tab file which was previously figures in the sequence of frames have been processed 

£nt2 ZTT^TV* f°? m% rcpre- If not, the loop 733 U continued until all jL^naTe 

*S&, y, ?L ■ , .3 J""" ««omp«ying «0 been processed, as described. In this manner, the in-be- 

tob > ^ fS?"^?* a *5 ^""^ » represented tween frames are formed and stored. It can be noted, m 

by the block 613. ^ described heremabove, the display represented by the block 729, that the limbs asudated 

2™ e j£* °f 66 m < )la ? ea,ed ' ■» *? oP^tor'* op- with each joint are obtained in conjimctionv^Se 

uon, either m terms of a skeleton of joints or as a full joint in each frame of the sequence, the motion of fee 

figure of tabs. A cursor can be conventkmally em- 65 limb being completely defined as described above, by 

ployed to identify the jomt whose accompanying limb is the matrix tnnTonnanon values of the assc«iated joint 

to be manipulated. Alternatively, the limb associated In accordance with a feature of the mventionTwhen a 

with the joint to be manipulated can be identified with generated sequence of animation is displayed.Thc op^a 
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tor can modify the motion oft limb during a continuous 
display of the sequence, This is achieved by accessing 
the parametric traWonnaxion matrix for the joint asso- 
ciated with the selected limb, and substituting, during 
display of the sequence, parameter components which 
are selected by the operator (preferably, by mampula* 
don of a joystick), and are used in place of the previ- 
ously stored parameter components of the joint during 
the frame sequence in question. In the present embodi- 
ment, at each frame of a sequence being displayed, the 
x, y and z components of a selected parameter (move, 
rotation or scale factor) of a selected joint are incre- 
mented by an amount which is proportional to the oper- 
ator selection of x, y and z values using the joystick. The 
routine for effecting this feature is illustrated in FIG. 8. 

The frame sequence to be reviewed and modified is 
identified (block 811). The joint and parameter to be 
modified are selected by the operator, as represented by 
block 812, As frames are displayed, the operator manip- 
ulates the joystick to indicate increments in the x, y and 20 
z components of the selected parameter of the selected 
joint The sensing of these increments is represented by 
the block 821. The matrix parameter components are 
modified in accordance with the sensed increments, as 
represented by the block 822. A frame is then displayed 25 
(block 823), in accordance with the display routine of 
FIG. 5. Determination is then made (diamond 824) as to 
whether or not the full sequence of frames has been 
displayed. If not, the next frame of the sequence is pro- 
cessed (block 825) by reentering block 821, and the loop 30 
is continued as each frame, with the modified limb, is 
displayed. The procedure can then be performed for 
any selected joint or parameter, under operator control. 
Also, it will be understood that both the old and new 
sequences of matrices for the selected joint can be saved 
so that the operator can decide what is to be used in the 
finally compiled frame sequence, 

The motion of a limb during sequence of frames can 
also be varied by specific modification of the interpola- 
tion curve for each component of each parameter of 40 
transformation matrix of the joint associated with the 
particular limb. For example, in the present embodi- 
ment, the interpolation curve can be displayed, such as 
in FIG. 10 which shows a curve for the x component of 
position ("move") fitted to four key frames. The com- 45 
puted in-between frame x positions are shown as hollow 
dots and the key frame positions are shown as solid dots. 
The solid curve is the original interpolation curve, and 
the dashed line curve represents an operator-modified 
curve which can be input, for example, via the data 50 
tablet Preferably, the old and new curves are both 
saved so that the motion of a particular limb can be 
modified until the operator is satisfied. It will be under- 
stood that similar operation can be performed for the x, 
y and z com pon ents of rotational orientation and scale 
factor, as well as for the other components of position. 

FIG. 9 illustrates the routine for accessing and modi- 
fying the interpolation curve as described. The operator 
first indicates the sequence of frames to be reviewed 
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The invention has been described with reference to a 
particular embodiment, but variations within the spirit 
and scope of the invention will occur to those skilled in 
the art. For example, it will be understood that key 
frames can be defined in terms of individual figures 
which have key poses in those frames, so that different 
frames will be considered key frames with regard to 
different figures. Also, the invention is not dependent 
upon any particular type of coordinate transformations 
or viewing transformations being used, and additional 
techniques, such as for handling clipping can be imple- 
mented, as is known in the art While the invention is 
described in terms of a monochrome system, it has ap- 
plication to color systems as well. It will also be under- 
stood that special purpose hardware implementations of 
various described portions of the system could be used 
if desired. Finally, it can be noted that means, such as 
television camera 65, can be used to obtain images of 
figures and basic positions and geometries of three -di- 
mensional figures can be developed using these images 
as an aid. 
I claim: 

1. A method for generating a sequence of video 
frames representative of three-dimensianal animation, 
comprising the steps of: 

storing a plurality of key frames, each including a 
common figure having one or more joints, each 
joint having associated therewith a set of vectors 
defining a limb, each joint being defined in rach 
key frame by operator-controllable parameters 
which determine in three dimensions the position, 
rotational orientation, and scale factors of a local 
coordinate system in which the limb vectors are 
placed; 

generating a plurality of in-between frames which 
include said common figure having one or more 
joints and associated limbs corresponding to the 
joints and associated limbs of said common figure 
in successive key frames, the parameters of the 
joints of an in-between frame being obtained as 
follows: 

(a) performing an interpolation, in each dimension, of 
the position, rotational orientation, and scale fac- 
tor, for the local coordinate system associated with 
a joint, such that the interpolated parameter values 
of the joint lie between the parameter values associ- 
ated with the corresponding joints of the succes- 
sive key frames; and 

(b) repeating step (a) for other joints of said common 
figure, 

2. The method as defined by claim 1, wherein the 
joints of each figure are arranged in a hierarchical or- 
der, and wherein the positional coordinates and rota- 
tional orientations of the local coordinate system for a 
particular joint are determined with respect to the local 
coordinate system of the next higher joint in the hierar- 
chy. 

3. The method as defined by claim 1, further compris- 



(block 911). The joint and parameter component to be 60 ing the step of operator-controlling the interpolation, 



modified are then identified (block 912) and the corre- 
sponding interpolation curve (see FIG. 7 routine) is 
displayed, as represented by the block 913. The opera* 
tor can then draw the modified interpolation curve (or 
input the information in any other desired way), as 
represented by the block 914. The modified curve val- 
ues can then be stored, and the sequence of frames re- 
viewed with the new curve, 



during generation of the in-between frames, so as to 
change the motion of a figure limb. 

4. The method as defined by claim 2, further compris- 
ing the step of operator-controlling the interpolation, 

65 during generation of the in-between frames, so as to 
change the motion of a figure limb. 

5. The method as defined by claim L further compris- 
ing the steps of determining the two-dimensional coor- 
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dinatet of said figure at viewed from an imaginary cam- rfv«.«^i...rl 

era perspective, and displaying the furore on a screeT Z derating a limb, each joint being defined 

ing the steps of deiermming the cvvo^ens*^ coot ti^T ™S5f^!£? 1 ^ ne ? ,ons P<*- 

dmates of said figure at viewed from an imSryam,- 5 fi ' factors of . 

era perspective, and displaying the ng^eTTsc^ ^ "o^ ty^ m wmch me l mb vecion 

dmate, of said figure asTwed from an imaginary^- m^SSil^S^ " 

era perspective, and delaying the figiireTTscre^ io STlff ™? "^^''s^ co™pondmg to 

mg the steps of determining the two-dimensional coor- ajAl, }~^Z^ ^ 
dinate, of said figure a, v«wed from an imSy^- w^L^^T?,?^ ^«*k**- 
era perspective, and displaying the figBrT^nTscre^ of ,^.^1^ ^T 8 , parameters of the joints 
9. The method u^Ledby clam?7 wheSSd H rl^^^T b) » «"erpola- 
step of controlling thTm^poU^cJmr^^^ ^ f m ~ h d " na » 10 °- of «** Position, rotation^ 
^ontroDing, during di^ofX fig^tauTal r^^l*** ^J^J f ° f * C se- 
quence of video t^TtaeUlmS^ c^mpt ^^t^flr* -* *»* .«* 
nentsof position. rotational or^tiXoTsc^rf^ P*™ 6 * values of the joint lie 
joint aaaocuUed with add limb. anoo - orsc " ,m 8 oftne "«wated with the 

quence of compt ^tZ^Z^^f ^ ron - 

It The method as defined by claim 9, wherein aid <* ^ next Jugher joint in the bierar. 

li Th! m«^\f^l2v^f^ M, J . . . 30 dumg generation of the in-between frames, so at to 

a£££Z£SL? ^ Whemn Chm » e *■ «>*» of • figure limb. 

operaior<ontrollmg of three-dnnenstonal eomponentt 18. Apparatus at defined by claim 16. father «»». 

^ Z&gZSffSgZZ? ^ " P^^ope^c^it^^ 

pnsmg the step of displaying a curve representative of a 19. Apparatus as defined by claim 15 father 

ZSETLfJ" interpolate of the prising »ie^deSn?th7r^i^^ 

EmS* ° r , ^ {K T 0f 1 ordina£ = of said Hg^Tviewed taTES? 

2£3 ^en^ofvS™^ ^ ^ 20 Apparatus as defined by claim 16. further com- 

U . «. . pnsmg means for deterntining the two-dimensional co- 

co^L«tSf,tS g CUrV , erepreS ? ltatlvc ° ft Perspective, and means for displaying the figure 

component of the three-dimensional interpolation of the 45 on a screen. "Bure 

E^b°!?^^ t £ 0 ' U 0r , SC * le factors of » 21. Appwitus as defined by claim 17, further com- 

fo^nSSe^^^! 0 ' ^ ^^l" 0 ™ 8 Pri*i»8 «««« for determining the two^meSomOc^ 

Jul^^uen^^^ ^ tob „ ^e^^^andmeansford^layingtheffi 

fr^es A ren^u^^S5- a ° f ^ 21 Aw * r,tns « defined "y claim 18, further coo- 

c^ml ° f ****«™o^ «««*non, priang mean, for determining the twosumenaonaTct 

_L,„ r. M . „ . . .. , . , ordinates of said figure as viewed from an unaginary 

^SSJSLTSLll f ^ taCh Penpective. and means for displaying thefigt^ 

tnauamg a common figure having one or more 55 on a screen. 8 

joints, each joint having associated therewith a set . . . . . 



60 



65 



09/26/2002, EAST Version: 1.03.0002 



. ••••• m n.u IMH «•<• «MM t*t* SStt* IMM at * tatii ftti «ii Alt 

usgo«go9i9Bi 



REEXAMINATION CERnHCATE 1785th) 
United States Patent n-i 

Stern 



[it] Bl 4,600,919 
[45] Certificate Issued Sep. 15, 1992 



[54] THREE DIMENSIONAL ANIMATION Badler, N. 1. el al. — Special Problems in Human Move- 

ment Simulation; ACM (1980). 

[75] Inventor: Garbad Stera, Brookville, N.Y. Burtnyk. N. et aL— Computer Graphics and Film Ani- 

mation; Can. Jem. et Op. Reg. and Info. Proceuing 

[73] A-ignee: Ne- T-kJ-j- fT.****. 2£fc. N. et a..-Co- W ^ener. ted Key Fr-te 
OW Westtwry. N.Y. Animatioa; Journal of SMPTE (Mar. 1971). 

j—.-^. Burtnyk, N. et al.— Towards a Computer Animating 

Na XWGlMJvL 26, 1991 ^oci^T^-EUROCOMP (May 1974)l 

Hackatborn, Ronald J.— Anima II. A 3-D Color Ani- 
, Certifies* far: 5*** Syttem; SIOGRAPH Proceedings, 1977. 

Patent Na: 4.400919 Newman, W. M. et ai— Principles of Interactive Com- 

Itsoed: JiTS im poter Graphici. Chapter 22, Three Dunensiooal Tram. 

Anol No, 40L9ftS fonnatiocs And Pertpective^(I979 ed.>. 

Filed: A«Tl9«2 0»DooneJl. T. J. et al.-ORAMPS-A Graphics W 

fuate Interpreter for Real-Time Iat^ctrve. 
Three-Dimensional Picture Editing and Atumatian, 

a - 0090 1/16 Computer Graphics, (Aug. 19S1>. 

[52 1 UA ^ - 340/725; 340/728; Reeves, W. T. Inbetweening for Computer Animaooo 

_ 340/747 Utilizing Moving Point Coimiaintt--Coinxxiier Granh- 

[58] HeW of Search « 3407700,723,725,728, ics, (Aug. 1981? «"™™»-^P^ ^ 

340/747, 744, 709, 724; 338/183 Sutherland, I. E— Sketchpad a Man-Machine Graphi- 
cal Coinmumcatkm System— AFIPS 1963. 
(56] References Cited Zwerg, & M.— Sailing— An Example of Computer 

U.S. PATENT DOCUMENTS $T *" *™^*>« AFIPS, May 

3,190.955 VI965 Manrer 338796 "Computer-Assisted Visual Communication**, Barry 

3,360t2 1/1961 Harrison. Ul 340/725 D«vid Messier, Ph.D. dttserubon, Computer Science, 

3,323089 8/1970 Harrison, til 340/723 University of Utah (19731 

3,383,628 6/1971 Harrison. Ill 340/723 ^ 

3,603.964 9/1971 Harrison. Ill 340/725 / ™ tt "? £ww*«r— Alvm Oberley 

3,700,792 10/1972 Harrison et aL 338/93 f571 ARSXPAf*r 

3,723.803 3/1973 Harrison et al. 340/723 1 J ABSTRACT 

3,729,129 4/1973 Fletcher et al 340/709 **** disclosure is directed to an improved method and 

Mi!*? 8 ! l /my « 340/703 apparatus for generating a sequence of video frames 

" ^ - 340/723 representative of three4iincns.onal animation. A pm- 

K /9W ££™ ^<y°'k«Y<rw~arestor^ 

uSmm mm K ~ ^ * ^T 00 *"* <^ more jomta, and «cb 

3.906,197 9/1975 Graver 364/321 8 E * ch JOmt U defined » ***** by opera- 

3.917.953 11/1975 Iauiya 230/349 tor-controllable parmmeters which determine the three- 

3.996,673 12/1976 Vont et aL 340/728 dimerisionaJ position, rotational orientation, and scale 

4.017.680 4/1977 Asderson et al. 338/242 factors of a local coordinate system in which the hmb 

4,045.789 871977 Bristow 340/725 vectors are placed. A plurality of in-between frames are 

*?MZ f?* 1 *** 1 364/410 generated, the in-between frames including the com. 

asawsBsr — -as ittisttsxxsssss 

4,119.744 2/1980 Stern 354/93 «"«ween frames are obtaa^ by raterpob^ fa three 

4,200J67 4/1980 Hill I] 340/103 " tmcpn cm> * the position, rotational o ri en tati on, and 

4.213,189 7/1980 Mueller ctaL -™~.Z 364/900 * cal€ **ctors of the corresponding joints of the key 

4J32J74 u/1980 Chang et sL 3647900 frame*- In the pre feiie d ensboliment, the joints of 

4033.631 11/1980 Mahler 358/183 figure are arranged in hierarchical order, and the poai- 

tiVAll Y}01\ 5^ 14(1/747 tk) °* J coordinates and rotationaJ orientatiooa of the 

4 l£% ISwS. - T ZZZ M J°?*°** «y«« ^ • P«rticulaT joint «e deter. 

4,4«9^i9 12/1914 Beckwtth tt ai. 3647322 mjaed with respect to the local coordmate system of the 

OTHER PUBLICATIONS ^ ^s^cr joint in the hierarchy. Also, the operator 

RftdlM- w t - .i ns * i t> *« can control the tnterpoUtioa during display of the in-be- 

Badler. N. I et ^-Digital Representatrves of Human tween frames, so as to change the motion of a figure 

Movement; ACM Computing Surveys (Mar. 1979). limb. 



09/26/2002, EAST Version: 1.03.0002 



Bl 4,600,919 
1 2 

REEXAMINATION CERTIFICATE 

ISSUED UNDER 35 US .C 307 &y of said generated sequence of from* so as to change 

the modem of a fart limb. 

THE PATENT IS HEREBY AMENDED AS 3 79. 7V method as defined by claim 2b\ wherein the 
INDICATED BELOW **** °> m arranged in a hierarchial order. 

and wherein the positional coordinates and rotational ori- 
Metter eacftosed to beery brackets [1 saacartd ia the '* tatio ** of the local coordinate system for a particular 
attest, bat baseeea Meted and is m toner » part of the >~ ^* n f ettrmxMt * ^tk respea m the heat coordinate 
***** — »r» friittH Ht» ff iMofaM 10 J***™ of the next hither joint in the hierarchy 

to^|»tea7^^ ^ ^^^^^ ia The method as defined i* claim 2* wherein said 

operojor-controlHng the interpolation comprises manipu- 
AS A RESULT OF REEXAMINATION. IT HAS Jtth>w *m*nittd ~ 

BEEN DETERMINED THAT: . - . . . . „ t . t 

15 ii. The method as defined in claim 2X whe r e a t the 

Claims 1-22 mix cancelled. j*M of each figure are arranged in a hMrarchieJ order. 

and wherein the positional coordinates and rotational on- 

^ » system of the next higher Joint in the hierarchy. 

23 The method a* ****** h»eh*i*i -k~i. ~~ Apparatus as defined in claim U, wherein said sets 

of rectors define thrte^imennonal hmhs. and further comprising means for applying said sets of rectors defining 
mmSu^^ said Um^ to thefts ^'generated in^rweVS 

" ^^^^^ 

tinuous rideo sequence to riew the motJn of the thre+dU M^^a/^aT^ * * 

mt ?T%l J!Li insu > t « x, ii. Apparatus as defined in claim 31 further comprising 

24. The method as defined by claim 23^rheretn satd means for modifying in-between frames of said sentence of 
displaying step ™J^*^™*i rfe two-dimensional jo >imn W mean jfar subsequently ren^nglhVmZdl 
coordinates of satd hmbs as newed from an imaginary fied sequence of frames. 

camera penpectire jnd displaying the figure as a amtinu- S4. Apparatus as defined by claim 32. wherein the joints 
° f mn /r0mg V m ° ^ im ^ <* «*" fit*** ** arranged in a hierarchical order, and 

wwo/tf^ptoji . - w - L wherein the positional coordinates and rotational oriento- 

2X The method as defined in claim 24. further comprise 3$ tions of the local coordinate system for a particular joint 
ing modifying in-between frames of said sequence of are determined with respect to the heal coordinate system 
frames, and subsequently rendering the modified sequence of the next higher joint in the hierarchy, 
off****. iJ. Apparatus as defined by claim $4, further compris- 

26. The method as defined by claim 23. further compris- ing means far operator-controlling the interpolation during 
ing modifying in-between frames of said sequence of 40 said display of said generated sequence of frames, so as to 
frames, and subsequently rendering the modified sequence change the motion of a figure limb. 

°f £? m £ * * * 36 - Apparatus as defined in claim 32, further comprising 

27. The method as defined in claim 26, wherein said means for operator-controlling the interpolation during 
modifying includes operator-controlling the interpolation said display of said generated sequence of frames, so as to 
during said display of said generated sequence of frames so 45 change the motion of a figure limb. 

as to change the motion of a figure limb. • • • • • 
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